Skip to content

TST/CLN: legacy pickle testing #3310

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 5 commits into from
Apr 12, 2013
Merged

TST/CLN: legacy pickle testing #3310

merged 5 commits into from
Apr 12, 2013
+255 −60

Conversation

jreback
Copy link
Contributor

@jreback jreback commented Apr 10, 2013

CLN:

  • standardized testing data to live under tests/data (for that set of tests),
  • added method util/testing.get_data_path to give you this path

TST:

  • added io/tests/test_pickle.py and io/tests/generate_legacy_pickles.py
    to generate legacy pickle files and provide comparsions
    just run python path_to_generate_legacy_pickles.py

    it will create a version of pickle for that python version, arch, and system, and store in pandas/io/tests/data/legacy_pickle/VERSION

  • sample pickles for windows-64/2.7/0.10.1 and linux/2.7/0.10.1 and linux/2.7/0.11.0 added

  • need to run this on various combinations to generate the reference library

  • allows backwards compatibility testing for these pickles

@jreback
Copy link
Contributor Author

jreback commented Apr 10, 2013

anybody know if py3 is supposed to be able to read py2 pickles?
(this is a pickle I created in py2.7 reading in 3.3)

Do we have to unicode any strings in the data first?

======================================================================
ERROR: test_read_pickles_0_10_1 (pandas.io.tests.test_pickle.TestPickle)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/virtualenv/python3.3_with_system_site_packages/lib/python3.3/site-packages/pandas-0.11.0.dev_a8cdf20-py3.3-linux-x86_64.egg/pandas/io/tests/test_pickle.py", line 42, in test_read_pickles_0_10_1
    self.compare(vf)
  File "/home/travis/virtualenv/python3.3_with_system_site_packages/lib/python3.3/site-packages/pandas-0.11.0.dev_a8cdf20-py3.3-linux-x86_64.egg/pandas/io/tests/test_pickle.py", line 26, in compare
    data = pickle.load(fh)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xf0 in position 14: ordinal not in range(128)

@ghost ghost mentioned this pull request Apr 11, 2013
Merged
jreback added 4 commits April 11, 2013 07:21
@jreback
CLN: moved all data for tests to /data sub-dir of the tests
ea6d721
@jreback
ENH: added legacy pickel generation/testing suppport
f600e20 
TST: added io/tests/test_pickle.py to test stored pickles
     added io/tests/generate_legacy_pickles.py to create legacy pickles
       on various arch and sytems (need contributions)
     added some sample pickles for 0.10.1/0.11.0
@jreback
BLD: fix setup.py to load correctly data files
76b0c54
@jreback
BUG: reading py2 pickles in py3 ok now
e71b09d
@jreback
Copy link
Contributor Author

jreback commented Apr 11, 2013

Turns out this does work, but you have to jump thru some hoops, so I updated the main pickle load method
Apparently numpy saves the dtype strings as strings and not unicode (I guess for compat)

from core/common

def load(path):
    try:
        with open(path,'rb') as fh:
            return pickle.load(fh)
    except:
        with open(path,'rb') as fh:
            return pickle.load(fh, encoding='latin1')

@ghost
Copy link

ghost commented Apr 11, 2013

does the other way round work?

@jreback
Copy link
Contributor Author

jreback commented Apr 11, 2013

no, can't read a py3 pickle in py2 (its protocol is 3, which py2 doesn't know about), raises a ValueError (which I am now catching in the comparators). I added a py3 pickle as well. Cross platform is fine as well.

wesm added a commit that referenced this pull request Apr 12, 2013
@wesm
Merge PR #3310
ba774f5
@wesm wesm merged commit e7a1a6b into pandas-dev:master Apr 12, 2013
@wesm
Copy link
Member

wesm commented Apr 12, 2013

Merged. thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
0.11
Development

Successfully merging this pull request may close these issues.
2 participants
@jreback @wesm